Increase in data, size, or compute can lead to sudden learning of specific capabilities by a neural network -- a phenomenon often called "emergence''. Beyond scientific understanding, establishing the causal factors underlying such emergent capabilities is crucial to enable risk regulation frameworks for AI. In this work, we seek inspiration from study of emergent properties in other fields and propose a phenomenological definition for the concept in the context of neural networks. Our definition implicates the acquisition of general structures underlying the data-generating process as a cause of sudden performance growth for specific, narrower tasks. We empirically investigate this definition by proposing an experimental system grounded in a context-sensitive formal language and find that Transformers trained to perform tasks on top of strings from this language indeed exhibit emergent capabilities. Specifically, we show that once the language's underlying grammar and context-sensitivity inducing structures are learned by the model, performance on narrower tasks suddenly begins to improve. We then analogize our network's learning dynamics with the process of percolation on a bipartite graph, establishing a formal phase transition model that predicts the shift in the point of emergence observed in our experiments when changing the data structure. Overall, our experimental and theoretical frameworks yield a step towards better defining, characterizing, and predicting emergence in neural networks.

Since at In-Context Learning (ICL), an ability generative LLMs make predictions based on the given that allows them to create predictors from labeled prompt (i.e., the context), there is a natural relationship examples. Few studies have explored the interplay between prompt engineering and ICL (illustrated in Figure between ICL and specific properties of functions 1). The IO prompting paper introduced InstructGPT, a it attempts to approximate. In our study, we use a model that was trained to follow instructions, which was one formal framework to explore ICL and propose a of the first works to popularize ICL. The term was originally new task of approximating functions with varying introduced in the GPT-3 paper (Brown et al., 2020).

This is frame-by-frame prediction with no temporal constraints. This is a frame-by-frame keypoint prediction on each animation frame. No temporal information is used. We show how the network is able to utilize the same keypoints across object instances and consistently predict keypoints across viewing angles, even when parts are occluded such as the back legs. Your browser does not support the video tag.